In view of the problem that ViBe (Visual Background extractor) algorithm sometimes produces “ghost” when detecting moving targets and the misdetection problem of moving targets caused by the interference produced by the algorithm in the target detection with dynamic background, an improved ViBe algorithm was proposed based on the techniques of three frame differencing and morphological post-processing carried on the key frames extracted by Color Layout Descriptor (CLD). Firstly, the video key frame images were extracted by CLD. Secondly, three frame differencing was performed on the selected key frame images. The background model containing the moving target was filled by the difference results, obtaining the real background image. Thirdly, the moving target was detected, so as to eliminate the “ghost”. Finally, the morphological processing technique with adaptive threshold was added in the updating stage of background model to eliminate the interference information in the dynamic background model. The experimental results show that the proposed algorithm has superiority in avoiding ghost and anti-dynamic background interference in moving target detection, and when the similar measurement threshold is selected from 0.67 to 0.72, the accuracy of the algorithm can be as high as 99.4%, indicating that the algorithm can ideally detect the position information of the moving target.